Investigating Content and Construct Representation of a Common-item Design When Creating a Vertically Scaled Test

According to the equating guidelines, a set of common items should be a mini version of the total test in terms of content and statistical representation (Kolen & Brennan, 2004). Differences between vertical scaling and equating would suggest that these guidelines may not apply to vertical scaling in the same way that they apply to equating. This study investigated how well the guideline of content and construct representation was maintained while evaluating two stability assessment criteria (Robust z and 0.3-logit difference). The results indicated that linking sets that were not totally representative of the full test forms produced different vertical scales than the linking sets that were most representative of the full test forms. The results also showed that large disparities in the composition of linking sets produced statistically significant differences in the growth patterns of the resulting vertical scales, but small disparities in the composition of linking sets produced very similar vertical scales. Overall, the Robust z procedure was a more conservative approach to flagging unstable items. INVESTIGATING CONTENT AND CONSTRUCT REPRESENTATION 3 Introduction The common-item design (CID) is a data collection plan widely used in creating a vertical scale. Examinees’ performance on the common items across test forms is used to indicate the amount of growth that occurs from grade to grade (Kolen & Brennan, 2004). Different decisions regarding the structure of the design and the composition of the linking item set may lead to different vertical scales (Camilli, Yamamoto, & Wang, 1993; Harris, 2007; Loyd & Hoover, 1980; Williams, Pommerich, & Thissen, 1998; Yen, 1986). The literature on test score equating provides some guidelines for constructing a test that includes common items as a method of collecting data (Kolen & Brennan, 2004). According to the guidelines, the set of common items should be a mini version of the total test in terms of content and statistical representation. Appropriately selecting common items for the linking set ensures that the common items represent the total test sufficiently. Potential common items are identified when adjacent test forms are constructed, but the common items that become part of the final linking set are those common items that are reasonably stable in difficulty across forms. The equating literature also provides several criteria for screening common items. Different criteria may result in different sets of linking items. The research on equating has produced helpful guidelines for selecting and screening common items, yet the differences between vertical scaling and equating would suggest that these guidelines may not apply to vertical scaling in the same way that they apply to equating. Through the equating process, the examinees’ location estimates are adjusted to account for differences in difficulty between the test forms and placed onto a common metric. In vertical scaling, the examinee groups that are administered the level tests are assumed to be different in INVESTIGATING CONTENT AND CONSTRUCT REPRESENTATION 4 ability. The set of test questions from one test form to the other are deliberately designed to assess different levels of achievement. In a review of the literature, Cook and Petersen (1987) concluded that when groups differ in level of ability, special care must be taken when selecting the set of common items for the anchor test. Content representativeness of the items is an important concern and can seriously affect conventional equating results (Cook, Eignor, & Taft, 1985; Klien & Jajoura, 1985). In the context of vertical scaling, since the examinee groups are expected to differ in their level of achievement and the test forms differ in difficulty level, shifts in construct and content specifications tested across test forms can occur simply by design. In equating when common items are screened for stability, since the two test forms are expected to be interchangeable, the item-difficulty estimates for common items are expected to be almost the same. However, in vertical scaling the item difficulty estimates from two test forms across adjacent grades are expected to differ somewhat. Common items should typically be easier for the students at the higher grade level and more difficult for the students at the lower grade level and this should be reflected in the item difficulty estimates. Background and Rationale Given the differences between equating and vertical scaling, this study investigated how well the guideline of content and construct representation was maintained in the context of creating a vertical scale while evaluating two stability assessment criteria. This was accomplished by analyzing data gathered using the CID illustrated in Figure 1. This CID is a variation of a design proposed by Sudweeks et al. (2008) and it involved constructing a separate test for two mathematical constructs (Geometry and Measurement) intended to assess achievement relative to objectives in the Utah Core Curriculum for several adjacent grade levels. INVESTIGATING CONTENT AND CONSTRUCT REPRESENTATION 5 Curricular Grade Level of Items G1 G2 G3 G4 G5 G6 G7 G8